ViGAT: Bottom-Up Event Recognition and Explanation in Video Using Factorized Graph Attention Network

In this paper a pure-attention bottom-up approach, called ViGAT, that utilizes an object detector together with Vision Transformer (ViT) backbone network to derive and frame features, head process these features for the task of event recognition explanation in video, is proposed. The ViGAT consists graph attention (GAT) blocks factorized along spatial temporal dimensions order capture effectively both local long-term dependencies between objects or frames. Moreover, using weighted in-degrees (WiDs) derived from adjacency matrices at various GAT blocks, we show proposed architecture can identify most salient frames explain decision network. A comprehensive evaluation study performed, demonstrating approach provides state-of-the-art results on three large, publicly available video datasets (FCVID, MiniKinetics, ActivityNet) a .